Electronic device and method of controlling electronic device

ABSTRACT

An electronic device is provided. The electronic device includes a display, a communicator wirelessly connected to a stylus pen, a processor, and a memory storing instructions that cause the processor to acquire motion information of the stylus pen, in response to the motion information matching with a plurality of motion commands, output a plurality of command images corresponding to the plurality of motion commands, respectively, based on a user feedback command to select one of the plurality of command images, determine the motion information as a motion command corresponding to the selected command image, output a tutorial image, acquire motion information of the stylus pen corresponding to the tutorial image, and determine the motion information of the stylus pen as a motion command corresponding to the tutorial image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application, claiming priority under§ 365(c), of an International application No PCT/KR2022/003043, filed onMar. 3, 2022, which is based on and claims the benefit of a Koreanpatent application number 10-2021-0034601, filed on Mar. 17, 2021, inthe Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic device including a stylus pen.More particularly, the disclosure relates to and electronic deviceincluding a stylus pen, a user terminal, and a method of controlling theelectronic device.

2. Description of Related Art

A user terminal (e.g., a mobile device) may be a portable device capableof being portable and having at least one function capable of performingvoice call and video call, a function of inputting or outputtinginformation, or a function of storing data.

As the functions of the user terminals are diversified, the userterminals are equipped with complex functions, such as taking photos orvideos, playing music files or moving pictures, playing games, receivingbroadcasts, wireless Internet, and the like.

In addition, for the sake of user convenience, the user terminals areimplemented also in a form that the operation of the user terminal iscontrolled using a stylus pen, such as a stylus pen.

The user terminal may perform an operation based on the motion of thestylus pen, but have a limitation in that the user terminal does notsmoothly recognize the motion of the stylus pen for each user.

Such a limitation may occur as a characteristic that occurs in anunconscious behavior that is not recognized even by the user.

On the other hand, since the types of unconscious behaviors are alldifferent between users, it is difficult for the user terminal torecognize the motions for each type of unconscious behavior in practice.

Therefore, in order to solve the above-described limitation, research isbeing actively conducted to increase the recognition rate of the motionof the user.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providean electronic device and a method of controlling the same that arecapable of more accurately recognizing the motion of a stylus pen usinguser feedback, thereby increasing the user convenience.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic device isprovided. The electronic device includes a display, a communicationmodule configured to be wirelessly connected to a stylus pen, aprocessor operatively connected to the display and the communicationmodule, and a memory operatively coupled to the processor, wherein thememory stores instructions that cause the processor to acquire motioninformation of the stylus pen, in response to the motion informationmatching with a plurality of motion commands, output a plurality ofcommand images corresponding to the plurality of motion commands,respectively, to the display, and based on a user feedback command toselect one of the plurality of command images, determine the motioninformation as a motion command corresponding to the selected commandimage.

The instructions may cause the processor to output a tutorial image tothe display, acquire motion information of the stylus pen correspondingto the tutorial image, and determine the motion information of thestylus pen as a motion command corresponding to the tutorial image.

The instructions may cause the processor to after acquiring the motioninformation of the stylus pen corresponding to the tutorial image,output a feedback message to the display, and determine the motioninformation of the stylus pen as the motion command corresponding to thetutorial image based on a user input corresponding to the feedbackmessage.

The instructions may cause the processor to determine a matching scoreof each of the plurality of motion commands based on a degree ofsimilarity between the motion information and each of the plurality ofmotion commands, and output, to the display, a plurality of commandimages corresponding to the plurality of motion commands in which thematching score exceeds a predetermined value.

The instructions may cause the processor to compare a degree ofsimilarity between a motion part of the motion information exceeding apredetermined ratio and each of the plurality of motion commands, andmatch the motion information with at least one of the plurality ofcommands.

The instructions may cause the processor to perform supervised learningusing the plurality of motion commands as reference data and using themotion information as input data, to match the motion information withthe plurality of motion commands.

The instructions may cause the processor to, based on an applicationexecuted in the user terminal, assign a first weight to each of theplurality of motion commands, and match the motion information with atleast one of the plurality of motion commands based on the motioninformation and the first weight.

The instructions may cause the processor to form matching data in whicheach of the plurality of motion commands corresponds to the motioninformation for each user account registered on the user terminal, andexecute one of the plurality of commands based on the matching data andthe motion information.

The instructions may cause the processor to assign a second weight to apart of the motion information based on the matching data, and recognizethe motion information assigned the second weight as one of the motioncommands.

In accordance with another aspect of the disclosure, a method ofcontrolling an electronic device is provided. The method includesacquiring motion information of a stylus pen, in response to the motioninformation being matching with a plurality of motion commands,outputting a plurality of command images corresponding to the pluralityof motion commands, respectively, to a display, and based on a userfeedback command to select one of the plurality of command images,determining the motion information as a motion command corresponding tothe selected command image.

The determining of the motion information as the motion commandcorresponding to the selected command image may include outputting atutorial image to the display, acquiring motion information of thestylus pen corresponding to the tutorial image, and determining themotion information of the stylus pen as a motion command correspondingto the tutorial image.

The determining of the motion information as the motion commandcorresponding to the selected command image may include after acquiringthe motion information of the stylus pen corresponding to the tutorialimage, outputting a feedback message to the display, and determining themotion information of the stylus pen as the motion command correspondingto the tutorial image based on a user input corresponding to thefeedback message.

The outputting of the plurality of command images to the display mayinclude determining a matching score of each of the plurality of motioncommands based on a degree of similarity between the motion informationand each of the plurality of motion commands, and outputting, to thedisplay, a plurality of command images corresponding to the plurality ofmotion commands in which the matching score exceeds a predeterminedvalue.

The determining of the motion information as the motion commandcorresponding to the selected command image may include comparing adegree of similarity between a motion part of the motion informationexceeding a predetermined ratio and each of the plurality of motioncommands, and matching the motion information with at least one of theplurality of commands.

The determining of the motion information as the motion commandcorresponding to the selected command image may include performingsupervised learning using the plurality of motion commands as referencedata and using the motion information as input data, to match the motioninformation with the plurality of motion commands.

The determining of the motion information as the motion commandcorresponding to the selected command image may include based on anapplication executed in the user terminal, assigning a first weight toeach of the plurality of motion commands, and matching the motioninformation with at least one of the plurality of motion commands basedon the motion information and the first weight.

The method of controlling the electronic device according to anembodiment may further include forming matching data in which each ofthe plurality of motion commands corresponds to the motion informationfor each user account registered on the user terminal, and executing oneof the plurality of commands based on the matching data and the motioninformation.

The executing of one of the plurality of commands based on the matchingdata and the motion information may include assigning a second weight toa part of the motion information based on the matching data, andrecognizing the motion information assigned the second weight as one ofthe motion commands.

In accordance with another aspect of the disclosure, an electronicdevice is provided. The electronic device includes stylus pen, and auser terminal configured to perform a communication with the stylus pen,wherein the user terminal is configured to acquire motion information ofthe stylus pen, in response to the motion information being matchingwith a plurality of motion commands, output a plurality of commandimages corresponding to the plurality of motion commands, respectively,to the display, and based on a user feedback command to select one ofthe plurality of command images, determine the motion information as amotion command corresponding to the selected command image.

The user terminal may be configured to output a tutorial image to thedisplay, acquire motion information of the stylus pen corresponding tothe tutorial image, and determine the motion information of the styluspen as a motion command corresponding to the tutorial image.

The electronic device and the method of controlling the same accordingto an embodiment can more accurately recognize a motion of a stylus penusing user feedback, thereby increasing user convenience.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIGS. 1A and 1B are views illustrating an operating state of a styluspen and a user terminal according to various embodiments of thedisclosure;

FIG. 2 is a control block diagram illustrating a stylus pen and a userterminal according to an embodiment of the disclosure;

FIG. 3 is a control block diagram illustrating a user terminal accordingto an embodiment of the disclosure;

FIG. 4 is a diagram illustrating a tutorial image according to anembodiment of the disclosure;

FIG. 5 is a diagram for describing an operation in which a user terminaldetermines a score of a gesture command based on a degree of similaritybetween motion information and a pre-stored motion command according toan embodiment of the disclosure;

FIGS. 6A, 6B, and 6C are diagrams for describing feedback imagesaccording to various embodiments of the disclosure;

FIG. 7 is a diagram for describing an operation of comparing a motionpart of motion information that exceeds a predetermined ratio with amotion command according to an embodiment of the disclosure;

FIGS. 8A and 8B are diagrams for describing an operation of matchingmotion information with a motion command based on user feedbackaccording to various embodiments of the disclosure;

FIGS. 9A, 9B, 9C, and 9D are diagrams for describing an operation ofmatching motion information with a plurality of predetermined motioncommands according to various embodiments of the disclosure;

FIG. 10 is a diagram for describing an operation of assigning a weightto a motion command corresponding to an application execution accordingto an embodiment of the disclosure;

FIGS. 11A, 11B, 11C, and 11D are diagrams for describing an operation ofassigning a weight to motion information according to variousembodiments of the disclosure;

FIG. 12 is a diagram for describing an operation of executing a motioncommand through matching data with motion information according to anembodiment of the disclosure; and

FIGS. 13 and 14 are diagrams illustrating flowcharts according tovarious embodiments of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

It will be further understood that the term “connect” or its derivativesrefer both to direct and indirect connection, and the indirectconnection includes a connection over a wireless communication network.

It will be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements,

In the specification, it will be understood that, when a member isreferred to as being “on/under” another member, it may be directlyon/under the other member, or one or more intervening members may inaddition be.

Although the terms “first,” “second,” “A,” “B,” etc. may be used todescribe various components, the terms do not limit the correspondingcomponents, but are used only for the purpose of distinguishing onecomponent from another component.

As used herein, the singular forms “a,” “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

Reference numerals used for method operations are just used forconvenience of explanation, but not to limit an order of the operations.Thus, unless the context clearly dictates otherwise, the written ordermay be practiced otherwise.

Hereinafter, the principles and embodiments of the disclosure will bedescribed with reference to the accompanying drawings.

FIG. 1A is a diagram illustrating an operation state of a stylus pen anda user terminal according to an embodiment of the disclosure.

FIG. 2 is a control block diagram of a stylus pen and a user terminalaccording to an embodiment of the disclosure.

Meanwhile, in the specification, an electronic device 1 may refer to adevice including a stylus pen 11 and a user terminal 10.

In the specification, the stylus pen 11 may be provided as a pen-shapedtool used to control the user terminal 10.

Referring to FIGS. 1A and 2 , a user may control the user terminal 10through an operation of directly tapping or dragging on a display of theuser terminal 10 using the stylus pen 11 or an operation of clicking abutton on the stylus pen 11.

The user terminal 10 may be implemented as various electronic devicesincluding a touch screen, such as a mobile phone, a smart phone, atablet personal computer (PC), a personal digital assistant (PDA), aportable multimedia player (PMP), a digital camera, etc. However, thedisclosure is not limited thereto.

The stylus pen 11 may include a clickable button. The user may controlthe user terminal 10 by clicking the button on the stylus pen 11 invarious ways. The method of clicking the button may include a method ofclicking the button once, a method of double clicking the button, amethod of long clicking the button, and the like but may not be limitedthereto.

The stylus pen 11 may receive an input of clicking the button.

The stylus pen 11 may transmit a control signal corresponding to thereceived input to the user terminal 10.

The stylus pen 11 may transmit the control signal to the user terminal10 using a communication channel formed between the stylus pen 11 andthe user terminal 10.

In addition, the stylus pen 11 may include a motion sensor capable ofdetecting a motion of a user. Here, the motion sensor may include atleast one of a geomagnetic sensor, a position sensor, an accelerationsensor, a gyroscope sensor, and a proximity sensor, but the disclosureis not limited thereto.

When the stylus pen 11 includes a motion sensor, a user may present apredetermined motion using the stylus pen 11.

FIG. 1B is an exploded perspective view illustrating a stylus penaccording to an embodiment of the disclosure.

Referring to FIG. 1B, according to various embodiments, a stylus pen 11may include a pen housing 116 constituting an outer appearance of thestylus pen 11 and an inner assembly inside the pen housing 116. In theillustrated embodiment, the inner assembly may include all of variouscomponents mounted inside the pen and may be inserted into the penhousing 116 in one assembly operation.

The pen housing 116 may have an elongated shape between a first end 116a and a second end 116 b, and may include an accommodation space 116 ctherein. The pen housing 116 may have an elliptical cross-sectionincluding a major axis and a minor axis, and may be formed in anelliptical column shape as a whole. The accommodation space 116 c of theelectronic device 101 may also have an elliptical cross-section tocorrespond to the shape of the pen housing 116. The pen housing 116 mayinclude a synthetic resin (e, g., plastic) and/or a metallic material(e.g., aluminum). According to an embodiment, the second end 116 b ofthe pen housing 116 may be formed of a synthetic resin material.

The inner assembly may have an elongated shape to correspond to theshape of the pen housing 116. The inner assembly may be largely dividedinto three parts along the longitudinal direction. For example, theinner assembly may include an ejection member 112 disposed at a positioncorresponding to the first end 116 a of the pen housing 116, a coil part113 disposed at a position corresponding to the second end 116 b of thepen housing 116, and a circuit board part 114 disposed at a positioncorresponding to the body of the housing.

The ejection member 112 may include a component for withdrawing thestylus pen 11 from the accommodation space 116 c of the electronicdevice 101. According to one embodiment, the ejection member 112 mayinclude a shaft 112 a, an ejection body 112 b disposed around the shaft112 a and constituting the overall appearance of the ejection member112, and a button part 112 c. When the inner assembly is fully insertedinto the pen housing 116, a portion including the shaft 112 a and theejection body 112 b may be surrounded by the first end 116 a of the penhousing 116, and the button part 112 c (e.g., 201 a in FIG. 2 ) may beexposed to the outside of the first end 116 a. A plurality of componentsnot shown, for example, cam members or elastic members, may be disposedwithin the ejection body 112 b to form a push-pull structure. In oneembodiment, the button part 112 c may be substantially coupled to theshaft 112 a to make a linear reciprocating motion with respect to theejection body 112 b. According to various embodiments, the button part112 c may include a button having a locking structure such that a usermay take out the stylus pen 11 using a fingernail. According to anembodiment, the stylus pen 11 may provide another input method byincluding a sensor for detecting a linear reciprocating motion of theshaft 112 a.

The coil part 113 may include a pen tip 113 a exposed to the outside ofthe second end 116 b when the inner assembly is fully inserted into thepen housing 116, a packing ring 113 b, a coil wound a plurality of timesand/or a pen pressure sensing part 113 d for acquiring a change inpressure according to the pressing of the pen tip 113 a. The packingring 113 b may include epoxy, rubber, urethane, or silicone. The packingring 113 b may be provided for the user of waterproofing anddustproofing, and may protect the coil part 113 and the circuit boardpart 114 from water immersion or dust. According to an embodiment, acoil 113 c may form a resonant frequency in a set frequency band (e.g.,500 kHz), and may be combined with at least one element (e.g., acapacitive element) to adjust a resonance frequency formed by the coil113 c in a certain extent of range.

The circuit board part 114 may include a printed circuit board 114 b, abase 114 a surrounding at least one surface of the printed circuit board114 b, and an antenna. According to an embodiment, the base 114 a may beprovided on an upper surface thereof with a substrate seating part 114 con which the printed circuit board 114 b is disposed, and the printedcircuit board 114 b seated on the substrate seating part 114 c may befixed. According to an embodiment, the printed circuit board 114 b mayinclude an upper surface and a lower surface, and a variable capacitancecapacitor or switch 114 d connected to the coil 113 c may be disposed onthe upper surface, and a charging circuit, a battery, or a communicationcircuit may be disposed on the lower surface. The battery may include anelectric double layered capacitor (EDLC). The charging circuit may bepositioned between the coil 113 c and the battery, and may include avoltage detector circuitry and a rectifier.

The antenna may include an antenna structure 114 i as shown in FIG. 1Band/or an antenna embedded in the printed circuit board 114 b. Accordingto various embodiments, the switch 114 d may be provided on the printedcircuit board 114 b. A side button 114 g provided on the stylus pen 11may be used to press the switch 114 d and may be exposed to the outsidethrough a side opening 116 d of the pen housing 116. The side button 114g is supported by a support member 114 h, and unless an external forceapplied to the side button 114 g, the side button 114 g may be restoredor maintained at a predetermined position with an elastic restoringforce provided by the support member 114 h.

The circuit board part 114 may include other packing rings, such asO-rings. For example, O-rings formed of an elastic material may bedisposed at both ends of the base 114 a to form a sealing structurebetween the base 114 a and the pen housing 116. In some embodiments, thesupport member 114 h may partially come in close contact with an innerwall of the pen housing 116 around the side opening 116 d to form asealing structure. For example, the circuit board part 114 may also havea waterproof and dustproof structure similar to the packing ring 113 bof the coil part 113.

The stylus pen 11 may include a battery seating part 114 e on an uppersurface of the base 114 a formed to seat a battery 114 f thereon. Thebattery 114 f that may be mounted on the battery seating part 114 e mayinclude, for example, a cylinder type battery.

The stylus pen 11 may include a microphone (not shown). The microphonemay be directly connected to the printed circuit board 114 b orconnected to a separate flexible printed circuit board (FPCB) (notshown) connected to the printed circuit board 114 b. According tovarious embodiments, the microphone may be disposed in a positionparallel to the side button 114 g in the longitudinal direction of thestylus pen 11.

A motion sensor in the stylus pen 11 may detect a motion of a user, andthe user terminal 10 may be controlled based on a signal correspondingto the detected motion of the user.

For example, when a camera application is running in the user terminal10, a zoom-in operation in the camera application may be performed basedon a motion of a user drawing a circle using the stylus pen 11.

Here, the stylus pen 11 may transmit a control signal to the userterminal 10 using short-range communication including Bluetooth,infrared, and the like.

In addition, the user terminal 10 may, in response to receiving a userinput, such as tapping or dragging on the touch screen, generate acontrol signal corresponding to the received user input.

According to an embodiment, the stylus pen 11 may include a battery.When the stylus pen 11 is inserted into a slot of the user terminal 10,the battery of the stylus pen 11 is connected to a power supply circuitinside the user terminal 10, so that the battery of the stylus pen 11may be charged. Meanwhile, according to another embodiment, the styluspen 11 may be charged through wireless charging with the user terminal10, even without being inserted into the slot.

Meanwhile, the operation described with reference to FIGS. 1A and 1B ismerely an embodiment of an operation between the user terminal 10 andthe stylus pen 11 included in the electronic device 1, and the operationof the stylus pen 11 and the user terminal 10 is not limited thereto.

FIG. 3 is a control block diagram illustrating a user terminal accordingto an embodiment of the disclosure.

Referring to FIG. 3 , a user terminal 10 according to the disclosedembodiment may further include an audio/video (A/V) input part 160 and amemory 170 as well as a sensing part 110, a communication module 120, acontrol part or controller 130, a user input part 140, and a displaypart 150.

The sensing part 110 may detect a state of the user terminal 10 or astate around the user terminal 10, and transmit the detected informationto the control part 130.

The sensing part 110 may include a motion sensor.

Specifically, the sensing part 110 may include at least one of ageomagnetic sensor (a magnetic sensor) 111, an acceleration sensor 112s, a temperature/humidity sensor 113 s, an infrared sensor 114 s, agyroscope sensor 115, a position sensor (e.g., a global positioningsystem (GPS)) 116 s, a barometric pressure sensor 117, a proximitysensor 118, or red, green, blue (RGB) sensor (an illuminance sensor)119, but the disclosure is not limited thereto. Since the function ofeach sensor may be intuitively inferred from the name thereof by aperson skilled in the art, detailed descriptions thereof will beomitted.

The sensing part 110 may sense a motion of the user terminal 10, abiometric signal of the user of the user terminal 10, a touch signalinput to the user terminal 10, and the like.

The communication module 120 may include one or more components forperforming communication between the user terminal 10 and the stylus pen11 or communication with an external device. For example, thecommunication module 120 may include a short-range wirelesscommunication module 121, a mobile communication module 122, and abroadcast reception module 123.

The short-range wireless communication module 121 may include aBluetooth communication module, a near field communication module, awireless local area network (WLAN) (e.g., wireless fidelity (Wi-Fi))communication module, a Zigbee communication module, an infrared(infrared Data Association (IrDA)) communication module, a Wi-Fi Direct(WFD) communication module, an ultra-wideband (UWB) communicationmodule, an Ant+ communication module, and the like, but the disclosureis not limited thereto.

The mobile communication module 122 transmits and receives wirelesssignals to and from at least one of a base station, an externalterminal, or a server on a mobile communication network. Here, thewireless signal may include various types of data according totransmission and reception of a voice call signal, a video call signal,or a text/multimedia message.

The broadcast reception module 123 receives a broadcast signal and/orbroadcast-related information from the outside through a broadcastchannel. The broadcast channel may include a satellite channel and aterrestrial channel. Depending on the embodiment, the mobile terminal100 b may not include the broadcast reception module 123.

Meanwhile, the communication module 120 may receive motion informationof the stylus pen 11 from the stylus pen 11 including a motion sensor.

In addition, the communication module 120 may be provided to receive adirection signal from an anchor device 3 that includes a directiondetection sensor.

An output part 151 is provided to output an audio signal, a videosignal, or a vibration signal, and may include the display part 150, asound output part 152, a vibration motor 153, and the like.

The display part 150 displays and outputs information processed by theuser terminal 10.

In addition, when the display part 150 is provided as a touch screen byforming a layer structure with a touch pad, the display part 150 mayalso serve as a stylus pen as well as an output device.

The display part 150 may include at least one of a liquid crystaldisplay, a thin film transistor-liquid crystal display, an organiclight-emitting diode, a flexible display, a three-dimensional (3D)display, or an electrophoretic display.

The sound output part 152 outputs audio data received from thecommunication module 120 or stored in the memory 170.

In addition, the sound output part 152 outputs a sound signal related toa function (e.g., a call signal reception sound, a message receptionsound, and a notification sound) performed by the user terminal 10. Thesound output part 152 may include a speaker, a buzzer, and the like.

The vibration motor 153 may output a vibration signal. For example, thevibration motor 153 may output a vibration signal corresponding to theoutput of audio data or video data (e.g., a call signal reception sound,a message reception sound, etc.). In addition, the vibration motor 153may output a vibration signal when a touch is input to the touch screen.

The user input part 140 refers to a device used for a user to input datafor controlling the user terminal 10. For example, the user input part140 includes a key pad, a dome switch, and a touch pad (e.g., a contactcapacitive method, a pressure resistance film method, an infraredsensing method, a surface ultrasonic conduction method, an integral typetension measurement method, a piezo effect method, etc.), a jog wheel, ajog switch, and the like, but the disclosure is not limited thereto.

The A/V input part 160 is provided to input an audio signal or a videosignal, and may include a camera 161, a microphone 162, and the like.

The camera 161 may obtain an image frame of a still image or a movingimage through an image sensor in a video call mode or a shooting mode.The image captured through the image sensor may be processed through thecontrol part 130 or a separate image processing part (not shown).

The image frame processed by the camera 161 may be stored in the memory170 or transmitted to the outside through the communication module 120.The camera 161 may be provided as two or more cameras 161 according tothe configuration of the user terminal 10.

The microphone 162 receives an external sound signal and processes thereceived sound signal into electrical speech data. For example, themicrophone 162 may receive a sound signal from an external device or aspeaker. The microphone 162 may use various noise removal algorithms forremoving noise generated in the process of receiving an external soundsignal.

The memory 170 may store a program for processing and controlling of thecontrol part 130, and may store input/output data (e.g., applications,content, time zone information of an external device, address books,etc.).

The memory 170 may include at least one type of storage medium among aflash memory type memory, a hard disk type memory, a multimedia cardmicro type memory, a card type memory (e.g., a secure digital (SD) or XDmemory), a random access memory (RAM), a static random access memory(SRAM), a read-only memory (ROM), an electrically erasable programmableread-only memory (EEPROM), a programmable read-only memory (PROM), amagnetic memory, a magnetic disk, and an optical disk. In addition, theuser terminal 10 may operate a web storage or a cloud server thatperforms a storage function of the memory 170 on the Internet.

Meanwhile, a plurality of motion commands may be stored in the memory170.

The motion command may refer to a command recognized by the userterminal based on a motion.

Programs stored in the memory 170 may be classified into a plurality ofmodules according to the functions thereof, for example, a userinterface (UI) module 171, a touch screen 172, a notification module173, and the like.

The UI module 171 may provide a specialized UI, a graphical userinterface (GUI), etc., that are linked with the user terminal 10 foreach application.

The touch screen 172 may detect a user's touch gesture on a touch screenand transmit information about the touch gesture to the control part130. The touch screen 172 according to the disclosed embodiment mayrecognize and analyze a touch code. The touch screen 172 may beconfigured as separate hardware including a controller.

In order to detect a touch or a proximity touch of the touch screen,various sensors may be provided inside or near the touch screen. Anexample of the sensor for detecting a touch of a touch screen includes atactile sensor. A tactile sensor refers to a sensor that detects a touchof a specific object at a certain extent of sensitivity felt by a humanor greater. The tactile sensor may detect various types of information,such as the roughness of a contact surface, the hardness of a contactobject, and the temperature of a contact point.

In addition, an example of the sensor for detecting a touch of a touchscreen may include a proximity sensor.

The proximity sensor refers to a sensor that detects the presence orabsence of an object approaching a predetermined detection surface or anobject existing in the vicinity of a predetermined detecting surfaceusing the force of an electromagnetic field or infrared rays withoutmechanical contact. Examples of the proximity sensor include atransmission type photoelectric sensor, a direct reflection typephotoelectric sensor, a mirror reflection type photoelectric sensor, ahigh frequency oscillation type proximity sensor, a capacitive typeproximity sensor, a magnetic type proximity sensor, an infraredproximity sensor, and the like.

A touch gesture of a user may include a tap, a touch and hold, a doubletap, a drag, a panning, a flick, a drag and drop, a swipe, and the like.

The notification module 173 may generate a signal for notifying theoccurrence of an event of the user terminal 10. Examples of the eventsgenerated in the user terminal 10 include call signal reception, messagereception, key signal input, schedule notification, and the like.

The notification module 173 may output a notification signal in the formof a video signal through the display part 150, may output anotification signal in the form of an audio signal through the soundoutput part 152, and may output a notification signal in the form of avibration signal through the vibration motor 153.

The control part 130 controls the overall operation of the user terminal10. For example, the control part 130 executes programs stored in thememory 170 to thereby control the sensing part 110, the communicationmodule 120, the output part 151, the user input part 140, the A/V inputpart 160, the memory 170, and the like.

The control part includes a memory for storing data regarding analgorithm for controlling the operations of the components of the userterminal 10 or a program that represents the algorithm, and a processor131 that performs the above described operations using the data storedin the memory. In this case, the memory and the processor may beimplemented as separate chips. Alternatively, the memory and theprocessor may be implemented as a single chip.

Meanwhile, the control part may be configured to include at least oneprocessor 131.

The processor 131 may acquire motion information of the stylus pen andmatch the motion information with a plurality of motion commands.

The motion information may refer to information obtained by acquiring amotion of the stylus pen.

The motion command may refer to a command for inputting a command to theuser terminal based on the motion of the stylus pen.

The processor 131 may output a plurality of command images correspondingto the plurality of motion commands, respectively, to the display.

The command image may refer to an image including the form of a motioncommand.

The processor 131 may, based on a user feedback command for selectingone of the plurality of command images, determine the motion informationas a motion command corresponding to the selected command image.

The processor 131 may output a tutorial image to the display.

The tutorial image may refer to an image output by the user terminal toguide a specific motion of the user in order to acquire motioninformation of the user.

The processor 131 may acquire motion information of the stylus pencorresponding to the tutorial image, and determine the motioninformation of the stylus pen as a motion command corresponding to thetutorial image.

That is, the processor 131 may output a tutorial image corresponding toa specific motion command, acquire user motion input subsequent to theoutput of the tutorial image, and match the user motion with thecorresponding motion command.

On the other hand, the processor 131, after acquiring the motioninformation of the stylus pen corresponding to the tutorial image, mayoutput a feedback message to the display.

The processor may, based on a user input corresponding to the feedbackmessage, determine the motion information of the stylus pen as themotion command corresponding to the tutorial image.

The processor 131 may acquire a user's motion subsequent to output ofthe tutorial image, and based on a user command that is input inresponse to a feedback image additionally output after the acquisitionof the user's motion, determine whether to match the user's motion witha gesture command.

The processor 131 may determine a matching score of each of theplurality of motion commands based on the degree of similarity of themotion information and each of the plurality of motion commands.

The matching score may refer to a standardized score determined by thedegree of similarity between motion information and each of a pluralityof predetermined motion commands.

For example, when the degree of similarity between the motioninformation and the motion command is high, a high matching score may berecorded, and when the degree of similarity between the motioninformation and the motion command is low, a low matching score may berecorded.

The processor 131 may determine the degree of similarity by comparingthe motion form of the motion information with the motion form of themotion command, and may assign a score to each of the correspondingmotion commands based on the degree of similarity.

The processor 131 may output, on the display, a plurality of commandimages corresponding to a plurality of motion commands whose matchingscores exceed a predetermined value.

Meanwhile, in determining the degree of similarity, the processor 131may compare the similarities between a motion part of the motioninformation exceeding a predetermined ratio and each of the plurality ofmotion commands.

That is, in comparing the motion information of the user with the motioncommand, the processor 131 may compare a part of the motion informationinstead of the entire motion information.

The processor 131 may perform supervised learning using the plurality ofmotion commands as reference data and using the motion information asinput data to match the motion information with the plurality of motioncommands.

Supervised learning is a method of machine learning for inferring afunction from training data. Training data generally includes attributesof an input object in the form of vectors, and may include desiredresults for each vector.

On the other hand, training data may refer to input data, and a desiredresult may refer to reference data.

In the disclosure, the user terminal may learn a large amount of motiondata using a plurality of motion commands as reference data to matchmotion data with a motion command.

The processor 131 assigns a first weight to each of the plurality ofmotion commands based on an application executed in the user terminal,and based on the motion information and the first weight, match themotion information with at least one of the plurality of motioncommands.

The first weight may refer to a weight assigned to a motion command,which increases the probability that motion data is recognized as aspecific motion command.

For example, when a specific application is running in the userterminal, the processor 131 may assign a greater weight to a motioncommand frequently used in the specific application such that thecorresponding motion command is determined as a motion command to matchwith the motion of the user.

The processor 131 may form matching data in which each of the pluralityof motion commands matches with motion information for each user accountregistered on the user terminal.

The matching data may refer to data including a correspondencerelationship between a user's motion and a motion command.

Meanwhile, such matching data may be determined for each user account.That is, matching data corresponding to a first user account may bedifferent from matching data corresponding to a second user account.

Accordingly, the processor 131 may execute one of a plurality ofcommands based on the matching data and the motion information.

The processor 131 may assign a second weight to a part of the motioninformation based on the matching data.

The second weight may refer to a weight applied to at least a part ofthe motion data in recognizing the motion data.

That is, the processor 131 may assign a greater weight to a specificpart of recognized motion information to thereby match the correspondingmotion information with the motion command.

In addition, the processor 131 may configure a software module thatperforms the above-described operations.

Specifically, the software module may include an application layer, aframework layer, a library layer, a kernel layer, and a hardware layer.

The application may be provided to draw at least one layer based on theresolution of a display area of the display.

The framework layer may include a plurality of programs.

In more detail, a window manager may be configured to, in response to astate change of the electronic device being identified through thesensor module, transmit information about the display area to theapplication.

In addition, the framework layer may include a stylus personalizationmanager Stylus Gesture Personalization Manager (SGPM).

The processor may perform the above-described operations through theSGPM.

The library layer, a hardware abstraction layer (HAL) may refer to anabstraction layer between a plurality of hardware modules included in ahardware layer and software of the electronic device.

The kernel layer may include various drivers for controlling varioushardware modules included in the electronic device. The kernel driverlayer may include a sensor driver including an interface module forcontrolling a sensor controller connected to a sensor.

At least one component may be added or omitted to correspond to theperformances of the components of the user terminal shown in FIG. 3 . Inaddition, the mutual positions of the components may be changed tocorrespond to the performance or structure of the system.

Meanwhile, some of the components shown in FIG. 3 may refer to asoftware component and/or a hardware component, such as a fieldprogrammable gate array (FPGA) and an application specific integratedcircuit (ASIC).

FIG. 4 is a diagram illustrating a tutorial image according to anembodiment of the disclosure.

Referring to FIG. 4 , a user terminal 10 may output a tutorial image 14on the display.

The user terminal 10 may receive a state of the stylus pen and motioninformation of the stylus pen.

In FIG. 4 , the motion information may refer to user behavior patterninformation and usage characteristic information of the user that arerecognized after the tutorial image 14 is output.

According to an embodiment, the user terminal 10 may output the tutorialimage 14 to the user and receive information about the user's repeatedmotions through the output of the tutorial image 14.

Specifically, as shown in FIG. 4 , the tutorial image 14 showing anupward swipe operation may be output on the display of the userterminal.

The tutorial image may include an image 14 capable of guiding the motionof the stylus pen and a description M4 thereof.

In FIG. 4 , the tutorial image 14 inducing the user to move the styluspen from top to bottom is output.

The user may, after the output of the tutorial image 14 to the userterminal, move the stylus pen upward to transmit the motion informationto the user terminal.

That is, the user terminal 10 may acquire the motion information of thestylus pen corresponding to the tutorial image 14,

That is, in FIG. 4 , the motion information of the stylus pen may bedetermined as a motion command corresponding to the tutorial image 14.

In FIG. 4 , a motion command for swiping from bottom to top isdisclosed, and the user terminal 10 may match the user motion with amotion command for swiping from top to bottom.

Meanwhile, the user terminal 10 may, after the output of the tutorialimage, receive gesture input information of the user to collect themotion information of the user, and determine whether the inputtedmotion information is normal.

Meanwhile, when the input motion information is incorrect, the userterminal 10 may induce the user to perform repeated actions severaltimes. Detailed descriptions thereof will be provided below.

In addition, the operation of matching the motion of the user with themotion command will be described in detail with reference to FIG. 5 .

FIG. 5 is a diagram for describing an operation in which a user terminaldetermines a score based on a degree of similarity between motioninformation and a pre-stored motion command according to an embodimentof the disclosure.

Meanwhile, the pre-stored motion command may include a swipe motioncommand, a circle motion command, a pointy motion command, and a shakemotion command.

Meanwhile, the user terminal 10 may determine a matching score of eachof the plurality of motion commands based on the degree of similarity ofthe acquired motion information and each of the plurality of motioncommands.

Referring to FIG. 5 , a motion command C5 and motion information I5 of auser are disclosed.

The user terminal may determine the matching score based on the degreeof similarity between motion information of the motion command C5 andthe motion information I5 of the user. According to an embodiment, theuser terminal 10 may compare a vertical length H5 of the motioninformation of the stylus pen with a vertical length of the motioncommand, and based on a horizontal length W5 of the motion informationof the stylus pen, determine the degree of similarity between the motioninformation of the motion command C5 and the motion information I5 ofthe user, and score the determined degree of similarity.

The user terminal may, in response to the degree of similarity betweenthe motion information and the corresponding motion command exceeding apredetermined score, determine that the corresponding motion informationmatches the motion command.

Meanwhile, in FIG. 5 , the operation of determining the degree ofsimilarity of the swipe operation has been described, but the degree ofsimilarity of each of the motion commands with the motion of the usermay also be determined by each method, and there is no limitation to theoperation.

FIGS. 6A to 6C are diagrams for describing feedback images according tovarious embodiments of the disclosure.

Referring to FIG. 6A, an operation in which, when the motion informationof the stylus pen inputted after the output of the tutorial imagematches the corresponding to the motion command, the user terminal 10outputs a feedback message is illustrated.

According to an embodiment, the user terminal may output a message M6 a,such as “Good job!”

FIG. 6A corresponds to a case in which the user motion informationsmoothly matches with the motion command intended by the tutorial image.

Accordingly, for the corresponding motion command, the user terminal maydetermine the motion information of the stylus pen input by the user asthe corresponding motion command.

Referring to FIG. 6B, FIG. 6B is a diagram illustrating an operation ofoutputting a feedback message in a case in which the motion informationof the user received after the output of the tutorial image does notsmoothly match with the motion command.

In this case, the user terminal may output a feedback message M6 brequesting the user to take action once again.

According to an embodiment, in a case in which the user inputs themotion of the stylus pen slowly, a message, such as “Move a littlefaster!,” may be output.

The user may, after the outputting of such a feedback message, inputmotion information of the stylus pen once again.

Meanwhile, when the user terminal 10 fails to match the motioninformation of the user with the motion command even after performingsuch an operation, a feedback message as shown in FIG. 6C may be output.

Referring to FIG. 6C, a feedback message output by the user terminal 10when it is difficult to match the motion information input by the userafter the outputting of the tutorial image with the motion command isillustrated.

The user terminal may output a feedback message M6 c that, when themotion information recognized by the user terminal after the outputtingof the tutorial image is difficult to match with the motion command,inquires to the user whether the user information input by the usercorresponds to the corresponding motion command, and outputs selectableinterfaces I6 c-1 and I6 c-2.

The user terminal 10 may output the motion command that is to match withthe motion information input by the user together with the feedbackmessage M6 c.

In FIG. 6C, an upward swiping motion is presented in the tutorial image,and a feedback message M6 c inquiring to the user whether an upwardswipe motion has been input as a user motion is output.

Meanwhile, the user terminal may, in response to a command correspondingto the feedback message being input by the user, match the motioninformation input by the user with the output motion command.

Specifically, the user terminal may, in response to the user selectingselectable interface “Yes, I like it” I6 c-1 on the interface as shownin FIG. 6C, match the motion information of the user with the motioncommand for swiping upward.

Meanwhile, the user terminal may, in response to the user selectingselectable interface “No. I will do it again” I6 c-2 on the interface asshown in FIG. 5C, allow motion information corresponding to the tutorialimage to be input once again.

Meanwhile, the operation described with reference to FIGS. 6A to 6C isonly an example of an output form of the feedback message of thedisclosure, and there is no limitation on the form of the feedbackmessage output by the user terminal.

FIG. 7 is a diagram for describing an operation of comparing a motioncommand with a motion of the motion information having a predeterminedratio according to an embodiment of the disclosure.

Referring to FIG. 7 , a user terminal 10 may compare the similaritiesbetween each of the plurality of motion commands and a motion part ofthe motion information that exceeds a predetermined ratio and may matchthe motion information with at least one motion command.

Specifically, when the user inputs motion information using the styluspen 11 as shown in FIG. 7 , the user terminal may divide thecorresponding motion into a part P71 and a part P72.

In FIG. 7 , the user terminal 10 may determine that a motion having thelongest lead distance in the user motion is a motion exceeding apredetermined ratio in the entire motion, and may determine the motioninformation based on the corresponding motion.

In the case of FIG. 7 , when considering both the part P71 and the partP72, the user terminal 10 may recognize both a circular motion and aswipe motion by the motion of the user, but may determine that the swipemotion has a long lead distance and exceeds a predetermined ratio in theentire motion.

That is, the user terminal 10 may determine the motion shown in FIG. 7as an action of swiping to the right, and thus may determine thecorresponding motion information to match with the swipe motion command.

Meanwhile, in this case, the user terminal 10 may output a commandimage, and based on a command of the user in response to the commandimage, match the motion information with the motion command as will bedescribed below.

FIG. 8A is a diagram for describing an operation of matching motioninformation with a motion command based on user feedback, according toan embodiment of the disclosure.

As described above, the user terminal 10 may acquire motion informationof the stylus pen and match the motion information with a plurality ofmotion commands.

Referring to FIG. 8A, it can be seen that the motion of the user matcheswith a circular motion command 181 and a swipe motion command 182, as inFIG. 7 .

Referring to FIGS. 7 and 8A, the part P71 of the motion of the user maycorrespond to the circular motion command 181 and the part P72 maycorrespond to the swipe motion command 182.

In this case, the user terminal may output a message M8 for requestingfeedback from the user and a plurality of command images (e.g., image ofcircular motion command 181 and image of swipe motion command 182)corresponding to the plurality of motion commands, respectively, on thedisplay as shown in FIG. 8A.

The user may input a feedback command for selecting one of the pluralityof output command images.

FIG. 8A shows a case of the user selecting the swipe motion command 182.

Referring to FIG. 8A, the user terminal 10 may determine thecorresponding user motion as a right swipe motion command D8.

That is, the user terminal 10 may, based on a user feedback command forselecting one of the plurality of command images, determine the motioninformation of the user as a motion command corresponding to theselected command image.

Meanwhile, the operation described with reference to FIG. 8A is merelyan embodiment for describing a user's feedback command for selecting aplurality of command images, and there is no limitation in the form ofthe image output by the user terminal or the type of the feedbackcommand.

FIG. 8B is a diagram illustrating a trajectory of a user on a userterminal according to an embodiment of the disclosure.

Referring to FIG. 8B, the user terminal may output a trajectory ML8 ofthe stylus pen to the user terminal through communication with thestylus pen.

The user terminal may output the trajectory of the stylus pen to informthe user of the type of motion information input by the user.

Based on the operation, the user may check whether the user has inputcorrect motion information.

The user may check the motion information itself and input motioninformation that may match with the command image.

FIGS. 9A to 9D are diagrams for describing an operation of matchingmotion information with a plurality of predetermined motion commands,according to various embodiments of the disclosure.

FIG. 9A is a diagram for describing an operation of determining motioninformation for a motion command corresponding to a swipe according toan embodiment of the disclosure.

Referring to FIG. 9A, a range a9 a may denote an allowable range for asingle motion in a direction perpendicular to a progress direction. Adistance b9 a may denote a minimum moving distance in the progressdirection. A range c9 a may denote a maximum allowable range from astarting point in the perpendicular direction.

A distance d9 a may denote a minimum distance for determining a motioncommand.

The user terminal may, when the motion information of the user includesa case of moving less than the range a9 a in a single action, a case ofmoving larger than the distance b9 a in the progress direction, a caseof moving from the starting point in the perpendicular direction by aseparation distance that is smaller than the range c9 a, and the like,determine the corresponding user motion as the motion commandcorresponding a swipe motion.

FIG. 9B is a diagram for describing an operation of determining motioninformation for a motion command corresponding to a circle according toan embodiment of the disclosure.

Referring to FIG. 9B, an angle a9 b may denote an angle formed by astart point and an end point at a center point of a path. A height w9 bmay denote the height of the motion information. A height h9 b maydenote the height of the gesture.

The user terminal 10 may, in a case of three points on the motioninformation not coinciding with a straight line or in a case of rotationin the user's motion direction, determine that the motion of the usercorresponds to a motion command corresponding to a circle.

In addition, even based on the aspect ratio (the ratio of the height w9b and the height h9 b) of the corresponding motion information, it maybe determined whether the corresponding motion information correspondsto a motion command of a circle.

FIG. 9C is a diagram for describing an operation of determining motioninformation for a motion command corresponding to an angle bracket(Pointy) according to an embodiment of the disclosure.

Referring to FIG. 9C, reference character M9 c may denote an inflectionpoint of a pointy gesture.

An angle a9 c may denote an angle formed by a line segment formed by anorigin and an inflection point. An angle b9 c may denote an includedangle of the inflection point. Meanwhile, in the case of an anglebracket, the user terminal may determine the pointy gesture based on thex-axis, the y-axis, and the position of the inflection point of themotion information.

In addition, the user terminal may, when the angle b9 c is greater than0 and less than 130 while moving in the motion direction, determine thecorresponding user motion as a pointy gesture.

In addition, the user terminal may determine whether the correspondingmotion information corresponds to the pointy motion command based on theangle a9 c or the angle obtained by subtracting the angle a9 c from 90degrees.

FIG. 9D is a diagram for describing an operation of determining motioninformation for a motion command corresponding to a shake according toan embodiment of the disclosure.

Specifically, FIG. 9D shows a plurality of inflection points O19 d, O29d, and O39 d of the motion of the user.

The user terminal 10 may detect a motion in a direction reverse to theprogress direction in the motion information, and when the includedangle of the motion is less than or equal to a specific angle, determinethat the corresponding motion information corresponds to a motioncommand corresponding to a shake.

In addition, the user terminal 10 may, when the corresponding inflectionpoint is detected more than a predetermined number of times in a singlepiece of motion information, determine that the corresponding motioninformation corresponds to the shaking motion command.

Meanwhile, the operation described in FIGS. 9A to 9D is only anembodiment in which the user terminal determines motion information tomatch with a predetermined motion command, and there is no limitation inthe method of the user terminal determining the acquired motioninformation.

FIG. 10 is a diagram for describing an operation of assigning a weightto a motion command corresponding to an application execution accordingto an embodiment of the disclosure.

Referring to FIG. 10 , the user terminal 10 may assign a weight to eachof a plurality of motion commands g10 based on an application A10executed in the user terminal.

In addition, the user terminal may match the motion information with atleast one of the plurality of motion commands based on the motioninformation and the weight.

In the case of FIG. 10 , the user terminal may, upon execution of aspecific application (e.g., the application A10) in which a pointygesture is frequently used, assign a weight to a corresponding motioncommand g′10 among the plurality of motion commands g10.

Here, the specific application may refer to a camera, an Internetapplication, or the like.

That is, the user terminal may control to assign a higher weight to thepointy motion command (e.g., the corresponding motion command g′10) inmatching the user motion with the gesture command during execution ofthe specific application such that the user motion is recognized as apointy gesture.

For example, when the motion of the user is recognized as a swipe motioncommand and a pointy motion command at a similar level, the userterminal may recognize the corresponding motion information as a pointycommand.

Meanwhile, the operation of the disclosure described with reference toFIG. 10 is only an embodiment of the disclosure, and there is nolimitation on the type of application to be executed and the motioncommand to which a weight is assigned.

FIGS. 11A to 11D are diagrams for describing an operation of assigning aweight to motion information according to various embodiments of thedisclosure.

The user terminal may assign a weight to a part of acquired motioninformation based on the matching data.

In addition, the user terminal may recognize the weighted motioninformation as one of the motion commands.

Hereinafter, various embodiments in which weights are differentlyapplied to each motion command will be described.

FIG. 11A is a diagram for describing an operation in which a weight isapplied to a user motion when the user motion matches with a pointymotion command according to an embodiment of the disclosure.

Referring to FIG. 11A, the user terminal may differently determineweights for recognizing a motion part W11 a and a motion part X11 a at atime of recognition of a pointy motion command according to matchingdata.

The user terminal may generally recognize the motions parts W11 a andX11 a in a proportion of 1:3.

However, the user terminal may, in a case of having matching data inwhich user feedback is reflected, recognize the motions parts W11 a andX11 a in a proportion of 1:5, so that the recognition of the motioninformation of the user may be improved.

FIG. 11B is a diagram for describing an operation in which a weight isapplied to a user motion when the user motion matches with an ellipticalmotion command according to an embodiment of the disclosure.

Referring to FIG. 11B, the user terminal may differently determineweights for recognizing a motion part W11 b and a motion part H11 b at atime of recognition of an elliptical motion command according tomatching data.

The user terminal may generally recognize the motion parts W11 b and H11b in a proportion of 4:10. However, the user terminal may, in a case ofhaving matching data in which user feedback is reflected, recognize themotion parts W11 b and H11 b in a proportion of 4:5; 10, so that therecognition of the motion information of the user may be improved.

FIG. 11C is a diagram for describing an operation in which a weight isapplied to a user motion when the user motion matches with anotherpointy motion command according to an embodiment of the disclosure.

Referring to FIG. 11C, when recognizing a pointy motion command, anglesformed by the motion information with the X-axis and the Y-axis may beconsidered according to the above-described matching data.

Specifically, the user terminal may acquire angles b11 c and a11 cformed with each of the x-axis and the y-axis, respectively, and whenthe angle formed with the y-axis is smaller than the angle formed withthe x-axis and when the difference between the two angles is greaterthan or equal to a specific value, assign a higher weight to the pointymotion command.

FIG. 11D is a diagram for describing an operation in which a weight isapplied to a user motion when the user motion matches with anothercircular motion command according to an embodiment of the disclosure.

Referring to FIG. 11D, when the above-described weight is applied andthe probability of identifying the pointy motion command increases, itmay be difficult for the user terminal to identify the correspondingmotion information as a circular motion command, even when the motioninformation is a completely closed circular motion.

Accordingly, the user terminal may form mapping data by assigning aweight P11 d when the circular motion is identified as being closed suchthat closed circular motion information is recognized as a circularmotion command even when the pointy motion command is weighted.

Meanwhile, FIGS. 11A to 11D are only an embodiment in which weights aregiven to motion information and motion commands in the disclosure, andthere is no limitation on the operation of assigning weights to motioninformation and motion commands.

FIG. 12 is a diagram for describing an operation of executing a motioncommand using matching data with motion information according to anembodiment of the disclosure.

The user terminal may form matching data in which each of the pluralityof motion commands matches with motion information for each user accountregistered on the user terminal.

Specifically, the user terminal may, upon motion information I11 of theuser being acquired, receive feedback from the user and match the motioninformation with a motion command, to thereby form matching data basedon the motion command matched with the motion information I11.

Meanwhile, the matching data is data for improving the recognition rateof motion for each user, and the user terminal may form the matchingdata for each user account.

Referring to FIG. 12 , the user terminal may, even with the same motioninformation being acquired, recognize the motion information I11 as amotion command C12-1 of motion information of swiping sideways in a caseof using matching data corresponding to a first user account ID1.

Meanwhile, even with the same motion information, the user terminal mayrecognize the motion information as a motion command C12-2 of motioninformation of swiping upward in a case using matching datacorresponding to a second user account ID2.

That is, the user terminal may form the matching data based on thefeedback of the user and recognize the motion data as a motion commandusing the matching data for each user account to thereby improve therecognition rate of the motion information of the user.

Meanwhile, the operation of the disclosure described with reference toFIG. 12 is only an embodiment of the disclosure, and there is nolimitation in the operation of the user terminal recognizing motion datausing matching data.

FIGS. 13 and 14 are diagrams illustrating flowcharts according tovarious embodiments of the disclosure.

Referring to FIG. 13 , the user terminal may acquire motion informationof the input device, in operation 1001.

In addition, the user terminal may determine the degree of similaritybetween the motion information and a plurality of motion commands, andmay determine a score of each motion command based on the degree ofsimilarity, in operation 1002.

Meanwhile, the user terminal may output a selection image of a motioncommand associated with the corresponding motion information, inoperation 1003.

Thereafter, the user may input a feedback for selecting thecorresponding motion command to the user terminal, and the user terminalmay match the motion information with the motion command based on thefeedback, in operation 1004.

Meanwhile, FIG. 14 is a diagram for describing an operation of matchingmotion information with a motion command using a tutorial image.

Referring to FIG. 14 , the user terminal may output a tutorial image, inoperation 1011.

The user terminal may, after the output of the tutorial image, acquiremotion information of the stylus pen, in operation 1012.

Thereafter, the user terminal may output a feedback message for thecorresponding motion, in operation 1013. Since the type and content ofthe feedback message have been described above, detailed descriptionsthereof will be omitted.

Meanwhile, the user terminal may match the motion information with themotion command based on a user input corresponding to the feedbackmessage, in operation 1014.

Meanwhile, the disclosed embodiments may be embodied in the form of arecording medium storing instructions executable by a computer. Theinstructions may be stored in the form of program code and, whenexecuted by a processor, may generate a program module to perform theoperations of the disclosed embodiments. The recording medium may beembodied as a computer-readable recording medium.

The computer-readable recording medium includes all kinds of recordingmedia in which instructions which may be decoded by a computer arestored, for example, a read-only memory (ROM), a random access memory(RAM), a magnetic tape, a magnetic disk, a flash memory, an optical datastorage device, and the like.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes is form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic device comprising: a display; acommunicator configured to be wirelessly connected to a stylus pen; aprocessor connected to the display and the communicator; and a memoryconnected to the processor and storing instructions which cause theprocessor to: acquire motion information of the stylus pen, in responseto the motion information matching a plurality of motion commands,output, to the display, a plurality of command images corresponding tothe plurality of motion commands, respectively, and based on a userfeedback command selecting one of the plurality of command images,determine the motion information as a first motion command correspondingto the selected command image.
 2. The electronic device of claim 1,wherein the instructions further cause the processor to: output, to thedisplay, a tutorial image, acquire motion information of the stylus pencorresponding to the tutorial image, and determine the motioninformation of the stylus pen as a second motion command correspondingto the tutorial image.
 3. The electronic device of claim 2, wherein theinstructions further cause the processor to: after acquiring the motioninformation of the stylus pen corresponding to the tutorial image,output, to the display, a feedback message, and based on a user inputcorresponding to the feedback message, determine the motion informationof the stylus pen as the second motion command corresponding to thetutorial image.
 4. The electronic device of claim 1, wherein theinstructions further cause the processor to: based on a degree ofsimilarity between the motion information and each of the plurality ofmotion commands, determine a matching score of each of the plurality ofmotion commands, and output, to the display, one or more command imagescorresponding to the plurality of motion commands in which the matchingscore exceeds a predetermined value.
 5. The electronic device of claim1, wherein the instructions further cause the processor to: compare adegree of similarity between a motion part of the motion informationexceeding a predetermined ratio and each of the plurality of motioncommands, and match the motion information with at least one of theplurality of motion commands.
 6. The electronic device of claim 5,wherein the instructions further cause the processor to: performsupervised learning using the plurality of motion commands as referencedata, and using the motion information as input data, match the motioninformation with the plurality of motion commands.
 7. The electronicdevice of claim 5, wherein the instructions further cause the processorto: based on an application executed in the electronic device, assign afirst weight to each of the plurality of motion commands, and based onthe motion information and the first weight, match the motioninformation with at least one of the plurality of motion commands. 8.The electronic device of claim 1, wherein the instructions further causethe processor to: form matching data in which each of the plurality ofmotion commands corresponds to the motion information for each useraccount registered on the electronic device, and based on the matchingdata and the motion information, execute one of the plurality of motioncommands.
 9. The electronic device of claim 8, wherein the instructionsfurther cause the processor to: based on the matching data, assign asecond weight to a part of the motion information, and recognize themotion information assigned the second weight as one of the plurality ofmotion commands.
 10. A method of controlling an electronic device, themethod comprising: acquiring motion information of a stylus pen; inresponse to the motion information being matching with a plurality ofmotion commands, outputting, to a display, a plurality of command imagescorresponding to the plurality of motion commands, respectively; andbased on a user feedback command selecting one of the plurality ofcommand images, determining the motion information as a first motioncommand corresponding to the selected command image.
 11. The method ofclaim 10, further comprising: outputting, to the display, a tutorialimage; acquiring motion information of the stylus pen corresponding tothe tutorial image; and determining the motion information of the styluspen as a second motion command corresponding to the tutorial image. 12.The method of claim 11, further comprising: after acquiring the motioninformation of the stylus pen corresponding to the tutorial image,outputting, to the display, a feedback message; and based on a userinput corresponding to the feedback message, determining the motioninformation of the stylus pen as the second motion command correspondingto the tutorial image.
 13. The method of claim 10, further comprising:based on a degree of similarity between the motion information and eachof the plurality of motion commands, determining a matching score ofeach of the plurality of motion commands; and outputting, to thedisplay, one or more command images corresponding to the plurality ofmotion commands in which the matching score exceeds a predeterminedvalue.
 14. The method of claim 10, further comprising: comparing adegree of similarity between a motion part of the motion informationexceeding a predetermined ratio and each of the plurality of motioncommands; and matching the motion information with at least one of theplurality of motion commands.
 15. The method of claim 14, furthercomprising: performing supervised learning using the plurality of motioncommands as reference data; and using the motion information as inputdata, matching the motion information with the plurality of motioncommands.
 16. The method of claim 10, wherein the motion informationcomprises user behavior pattern information and usage characteristicinformation of a user that are recognized after a tutorial image isoutput.
 17. The method of claim 10, further comprising: outputting atrajectory of the stylus pen to inform a user of a type of the motioninformation input by the user.
 18. The method of claim 11, wherein thetutorial image comprises a guide image output by the electronic deviceto guide a specific motion of a user.
 19. The method of claim 18,further comprising: outputting the guide image for guiding the specificmotion together with a description of the specific motion.